Max-margin Latent Dirichlet Allocation for Image Classification and Annotation

Much work in image classification and labeling uses topic models (e.g. LDA [1]), which are a class of powerful tools originally proposed in text modeling and have gained much popularity in computer vision recently. Despite the success of topic models in visual recognition, we believe there are some limitations of the way that topic models are used in computer vision. First of all, most topic models unsupervised. This means the topics discovered by topic models are not necessarily the ones used for discriminative tasks, such as image classification. To address this issue, several supervised variants of topic models have been developed. But the limitation of those models is that most of them assume the “bag-of-words” image representation, i.e. an image is represented by a collection of unordered feature descriptors computed from small local patches. Although the “bag-of-words” representation has been proven successful, other more holistic image representations (e.g. GIST [2]) have been shown to be powerful in many applications too. It is desirable to have the best of both worlds and design a model that can exploit both types of feature representation. In this paper, we propose the max-margin latent Dirichlet allocation (MMLDA), a variant of MedLDA [3]. We introduce two different versions of MMLDA, called MMLDAc for image classification, and MMLDAa for image annotation. MMLDAc is based on MedLDA. The main difference is that MedLDA only uses the latent topics as the feature vector for classification, while MMLDAc uses latent topics together with any other image features. This extension allows MMLDAc to make use of image features (e.g. GIST) that cannot be easily represented as bagof-words. MMLDAa is an extension of MMLDAc for image annotation. In image annotation, the goal is to choose a set of annotation terms (also called tags) to describe an image. Since an image can be associated with more than one tag, image classification is a multi-label classification. In MMLDAa, various tags are implicitly coupled by the latent topics defined in the model. Training MMLDAa results in topic representations that are suitable for predicting those tags. MMLDAc: We use x to denote an image. We use w to denote the bag-ofwords representation of x, e.g. w can be obtained by vector-quantization of SIFT descriptors. The topic assignment of the words in the document is denoted by z. We assume a linear discriminative function of the form F(y,z,w,x,η) = η> y f (z,w,x). Note the definition of F(·) is similar to that in MedLDA. In fact, if we assume f (z,w,x) = z̄ = 1 N ∑ N n=1 zn, we can recover F(·) in MedLDA. So the definition of F(·) in MMLDA is a strict generalization of that in MedLDA. One important thing to remember is that since z is not observed, f (z,w,x) is actually a random vector implicitly defined by the distribution on Z. We assume f (z,w,x) is a concatenation of two sub-vectors f (z,w,x)= cat(z̄;g(w,x)), where g(w,x) is a vector defined on w and x, z̄ is defined as z̄ = 1 N ∑ N n=1 zn similar to sLDA and MedLDA, cat(a;b) denotes the concatenation of two vectors a and b. Notice that we do not have any assumption on the form of g(w,x), it can be any feature vector extracted from the image, e.g. histogram of words, GIST descriptors, or both. Similarly, we assume ηy is also a concatenation of two sub-vectors ηy = cat(ζy;νy), so that η> y f (z,w,x) = ζy >z̄ + νy>g(w,x). Fig. 1 (a) shows a graphical illustration of MMLDAc. Similar to MedLDA, we learn the model parameter by solving an optimization problem as follows: